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^ DEVICE FOR RECORDING INFORMATIO N S DIFFERENT MODES \^ 



Field of the Invention 

The present invention relates to a device for 
recording information of the type mentioned in the 
preamble to the appended claim 1 . 
Background of t hp Invention 

In recent years, the use of computers has increas- 
ed among many very geographically mobile occupational 
groups. One reason for this is that one can work more 
efficiently by having direct access to a large amount 
of information while at the same time having good cap- 
abilities for processing new information simply and 
efficiently. 

For this purpose, various types of portable com- 
puters have been developed. Equipped with suitable soft- 
ware, these types of computers provide excellent capabi- 
lities for processing and managing information in text 
as well as image format. 

A drawback of these portable computers is that 
inputting information by the intermediary of a key- 
board is often time-consuming in comparison with the 
time required for processing the information once it has 
been fed into the computer. In order to achieve faster 
and more efficient inputting, there have been attempts 
at finding simpler ways of feeding the information into 
the computer. In some cases, the text which is to be fed 
into the computer is replaced by a numerical code, for 
instance in the case of vehicle inspections. However, 
this requires more of the user, who has to keep track of 
several codes, which increases the risk of input errors. 

In the daily work of many occupational groups there 
is a need for a simple way of putting together image and 
text information. Examples of this include various types 
of service and maintenance staff who work from some type 
of protocol and who wish to verify that a certain opera- 
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tion has been carried out; policemen at the site of an 
accident who wish to gather image data as well as per- 
sonal data concerning witnesses, etc.; and researchers 
and other individuals whose work involves information 
retrieval and who wish to put together text and images 
from a number of different written documents. 

In order to achieve more efficient inputting of 
information in these and similar situations one could 
use other input units, such as a scanner which records 
text information from a surface when it is passed over 
the same and a digital camera which takes pictures and 
stores them digitally so that they can be transferred 
to a computer for processing. 

Scanners of both the stationary and the portable 
type are available. Stationary scanners are preferably 
used in connection with the recording of whole pages with 
printed text and image information, the scanner being 
passed over the page at a constant speed. The information 
is recorded by means of a light-sensitive sensor, which 
in stationary scanners often consists of a line sensor. 

However, in most of the above situations, rather 
than the recording of a whole page of information a more 
selective recording of information is desired and, more- 
over, the scanner must also be easy to carry. For this 
reason, scanners of the hand-held type are much more 
interesting in these contexts. The sensors in these scan- 
ners can be either of the line type or of the two-dimen- 
sional type. 

US 4,949,3 91, for example, discloses a hand-held 
scanner for recording text. The scanner has a two-dimen- 
sional sensor, which records images of the underlying 
surface when the scanner is passed over the same. 

Another hand-held scanner is described in US 
5,301,234. This scanner utilises a line sensor which 
records a sequence of vertical "slices" of the characters 
and of the spaces between them when the scanner is passed 
over the characters on the substrate. The "slices" are 
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stored in the scanner as a digital bitmap. To enable the 
accurate recording of characters, the scanner is provided 
with position means in the form of a wheel which must be 
in contact with the underlying surface in connection with 
the recording of the characters to record the position of 
the scanner relative to the surface. 

A drawback of using a computer, a scanner, and a 
digital camera at the same time is that the equipment 
becomes bulky and the user has to handle several units. 

Thus, there is a need for equipment which is easy 
to handle, enabling fast and reliable recording of selec- 
tive text and image information in situations of the kind 
described above in a format facilitating computer pro- 
cessing of the same at a later time. 
Summary of the Invention 

The object of the present invention is to provide a 
solution to the above-mentioned problems by providing an 
improved device for recording information in text and 
image format . 

This object is achieved according to the invention 
by a device having the features recited in the appended 
claim 1, with preferred embodiments in the appended 
claims 2-14. 

The invention is based on the understanding that 
with a light-sensitive sensor with a two-dimensional 
sensor surface as the starting-point, a text recording 
function, i.e. a "scanner function", as well as an image 
recording function, i.e. a "camera function", can be 
easily created in one and the same device. More 
specifically, the device is adjustable between a first 
mode, in which the device is adapted to essentially abut 
against and be passed over a surface for imaging the same 
by means of a plurality of images, and a second mode, in 
which the device is adapted to image an object located at 
a distance. 

The device can be adjusted by the user or automati- 
cally by the device if, for example, it detects that the 
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distance to an object within its field of vision changes 
more than a predetermined value. The adjustment involves 
the adaptation of the device in some manner, e.g. physi- 
cally or with respect to software, to make it suitable 
5 for use in the mode in question. 

The first mode, or the "scanner mode", is intended 
for recording information located on a surface over which 
the device can be passed. The surface need not be plane. 
It is sufficient that it is even enough to enable the 

10 device to be moved in contact with the surface or direct- 
ly adjacent to the same. The information, which can be 
said to be "two-dimensional" since it is located on a 
surface, can be composed of character sequences, e.g. one 
or several words or one or several sentences selected 

15 from a longer text. The characters can be letters, num- 
bers, or other written characters or symbols. It is pre- 
ferably stored in character-coded format. However, the 
information may also be composed of image information 
which is scanned by passing the device over the surface 

20 and recording a plurality of images of the image informa- 
tion . 

The second mode is preferably used for recording 
"three-dimensional" information, i.e. when one is requir- 
ed to take pictures of objects. Preferably, in this mode, 
25 individual pictures are taken which are stored in image 
format. Naturally, this mode can also be used for taking 
pictures of surfaces from a distance, which are stored in 
image format . 

Preferably, the same sensor is employed in both the 
30 first and the second mode, but it is also possible to use 
one sensor for each mode. In the latter case, the adjust- 
ment would involve changing the beam path in the device 
so that the imaging takes place with the aid of one sen- 
sor in the first mode and with the aid of the other sen- 
3 5 sor in the second mode. The beam path can be changed by 
adjusting one or more components in the device. Alterna- 
tively, a first opening with a first fixed beam path can 
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be used in the first mode, and a second opening with 
a second fixed beam path can used in the second mode. 
Adjusting between the modes then comprises an adjustment 
of which sensor is activated for imaging. 

It should be noted in this context that a two-dimen- 
sional sensor surface refers to the fact that the sensor 
surface must be able to image a surface with a matrix of 
pixels so that images with overlapping contents can be 
recorded . 

Preferably, in the first mode, images with partially 
overlapping contents are recorded. The overlapping con- 
tents are used to put together the images to form a com- 
posite image, in which there is no duplication of con- 
tent. Suitably, the putting- together is carried out with 
the aid of a signal -processing unit which, for example, 
may contain software for this purpose. By virtue of the 
fact that the overlapping content is used for putting 
together the images, the device need not contain any 
means, e.g. wheels, for recording the position of the 
device relative to the surface which is being imaged or 
for measuring the distance between the images. Moreover, 
the device can be moved at varying speeds while still 
permitting the images to be put together into a composite 
image . 

The putting-together of the images is preferably 
carried out horizontally as well as vertically. This has 
the advantage that the device can be held at different 
angles and even be turned while being passed over the 
surface and yet the images can be put together well 
enough that the characters in the composite image can be 
identified and OCR processed. 

The signal -processing unit preferably comprises 
software for identifying characters in the composite 
image and for storing the same in the device in charac- 
ter-coded format, preferably ASCII code. The character 
identification is advantageously carried out with the aid 
of a neural network. Storing in character-coded format 



WO 99/60515 



PCT/SE99/00715 



6 

requires considerably less memory space than if the same 
amount of information is stored, for example, as a bitmap 
image . 

In order to project onto the sensor surface a clear 
image of the information to be recorded, i.e. the surface 
over which the device is passed in the first mode, or of 
the object located at a distance in the second mode, the 
device preferably comprises at least one lens means, 
which is adjustable between a first position in the first 
mode and a second position in the second mode. In this 
way, two foci are provided for the device, i.e. two dif- 
ferent imaging distances each providing a sharp image on 
the sensor surface. One focus is preferably used in the 
first mode where an object is located adjacent to the 
device and- the other is- preferably used in the second 
mode where an object is located at a distance from the 
device. The second position can suitably allow imaging of 
objects at distances of between 0.5m and infinity. The 
different foci enable imaging of surfaces or objects at 
different distances from the device. The lens means may 
comprise one or more adjustable lenses. The device may 
contain one or more fixed lenses in addition to the one 
or more adjustable lenses. This embodiment is intended 
in the first place for the case where the device has a 
single light-sensitive sensor which is common for both 
modes . 

In another embodiment, the device comprises a lens 
means, the position of which is variable for providing 
a variable focus or an autofocus function. This enables 
sharp imaging of an object located at an optional dis- 
tance . 

Furthermore, in an advantageous embodiment, the 
device is adapted to effect the imaging in the first mode 
with a lower resolution than the imaging in the second 
mode. This can, for example, be achieved in the first 
mode by making passive some of the sensor elements of the 
sensor or by saving only a part of the sensor values 
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which are recorded by the sensor, or by using only a part 
of the saved sensor values in the subsequent processing 
of the images, such as in connection with the putting- 
together. The advantage of this embodiment is that it 
5 is possible to achieve faster, yet reliable, putting- 
together of the images in the first mode if their reso- 
lution is lower. 

Moreover, the device advantageously comprises a 
transceiver for wireless communication with an external 

10 unit. In this way, information can be transferred between 
the device and, for example, an external computer. The 
transceiver can be an IR transceiver, a mobile radio 
transceiver, or some other suitable transceiver. 

To make the device easy to handle in all types of 

15 recording .situations it is suitably of the hand-held 

type. This also expands its field of application consi- 
derably . 

In a preferred embodiment, the device also comprises 
identification means for identifying the extent of the 

20 imaging. The identification means may, for example, com- 
prise a display mounted on the device, or one or several 
luminous spots projected from the device onto the object 
for defining the extent of the imaging. By the fact that 
the user gets an idea of what the image of the object 

25 will look like once it is recorded, the chances of 

obtaining an accurate imaging result will be substantial- 
ly better. 

Brief Description of the Drawings 

The invention will be described in more detail below 

3 0 with reference to the accompanying schematic drawings 

showing, by way of example, a presently preferred embodi- 
ment of the invention. 

Fig. 1 schematically shows a first embodiment of 
a device according to the invention; 

35 Fig. 2 is a block diagram of the electronic cir- 

cuitry in an embodiment of a device according to the 
invention ; 
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Fig. 3 is a flowchart showing how the device is 
intended to function in the first mode; 

Fig. 4 is a flowchart showing how the device is 
intended to function in the second mode; 
5 Fig. 5 schematically shows a second embodiment of 

a device according to the invention. 

Description of a Preferred E mbodiment 
Structure of the Device 

In the embodiment of the device according to the 

10 invention shown in Fig. 1, it comprises a casing 1 hav- 
ing approximately the same shape as a conventional high- 
lighter pen. One short side of the casing has a window 2. 

In the first mode of the device, the so-called scan- 
ning mode, the window 2 is intended to abut against or be 

15 held directly adjacent to a surface which one wishes to 
image. This surface can, for example, consist of a sheet 
of paper from which text is to be recorded. In the second 
mode of the device, the so-called camera mode, the window 
is intended to be directed at an object which is located 

20 at a distance and which one wishes to image. 

The dimensions of the window 2 determine the size of 
the surface which can be imaged in the first mode. In the 
second mode, the dimensions of the window 2 only consti- 
tute a partial factor when determining which objects can 

25 be recorded. Another partial factor of equal importance 
in the second mode is the distance to the object which 
the user wishes to image by means of the device. 

The window 2 forms an acute angle to the longitudi- 
nal axis of the device so that in the first mode the user 

30 is guided to hold the device at a predetermined angle to 
the underlying surface. Furthermore, the window 2 is 
somewhat recessed in the casing 1 so that is does not 
wear against the underlying surface when the device is 
used in the first mode. 

35 The casing 1 essentially contains an optics part 3, 

an electronic circuitry part 4, and a power supply 5. 
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The optics part 3 comprises a light -emit ting diode 
(LED) 6, an adjustable lens system 7, and a light-sensi- 
tive sensor 8 which constitutes the interface with the 
electronic circuitry part 4 . 

The LED 6 is preferably used in the first mode where 
its task is to illuminate the part of the surface which 
is currently located under the window. A diffuser 9 is 
mounted in front of the LED 6 for diffusing the light. In 
the second mode, the object being imaged is preferably 
illuminated by means of other external light sources in 
the vicinity of the object. 

The task of the lens system 7 is to project as 
accurately as possible an image of the surface located 
under the window 2, or of the object located within the 
field of vision of the window on the light-sensitive 
sensor 8. In connection with the adjustment between the 
two modes, the lens system 7 is moved between two posi- 
tions so that two different foci are provided. Fig. 1 
shows the lens system 7 in the first position A. The 
second position B is indicated, by way of illustration, 
by dashes extending from the lens system. The lens system 

7 can be moved in the same way as in a camera with two 
foci. Moreover, if required, the lens system 7 will 
reduce the image so that it will fit on the light-sen- 
sitive surface of the sensor 8. 

In this example, the light-sensitive sensor 8 com- 
prises a two-dimensional, square CCD unit (CCD = charge 
coupled device) with a built-in A/D converter. Such sen- 
sors are commercially available. In this case, the sensor 

8 is mounted at a small angle to the window 2 and on its 
own printed circuit board 11. 

The power supply to the device is obtained from a 
battery 12 which is mounted in a separate compartment 13 

in the casing . 

The block diagram in Fig. 2 schematically shows the 
electronic circuitry part 4. This comprises a processor 
20, which by the intermediary of a bus 21 is connected 
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to a ROM 22, in which the programs of the processor are 
stored, to a read/write memory 23, which constitutes the 
working memory of the processor and in which the images 
from the sensor as well as the characters identified and 
5 interpreted in the first mode are stored, to a control 
logic unit 24, as well as to the sensor 8, the lens sys- 
tem 7, and the LED 6. 

The control logic unit 24 is in turn connected to 
a number of peripheral units, comprising a display 25, 

10 which is mounted in the casing, an IR transceiver 2 6 for 
transferring information to/from an external computer, 
buttons 27, by means of which the user can control the 
device, a tracer LED 2 8 which emits a light beam, making 
it easier for the user to follow the text, as well as an 

15 indicating device 29, e.g. a couple of LEDs, indicating 
which mode the device is in and whether or not recording 
is taking place. Control signals to the memories, the 
sensor, and the peripheral units are generated in the 
control logic unit 24. The control logic also handles 

2 0 generation and priorit isat ion of interrupts to the pro- 
cessor. The buttons 27, the IR transceiver 26, the dis- 
play 25 and the tracer LED 28, and the LED 6 are accessed 
by the processor writing and reading in the register in 
the control logic unit. The buttons 27 generate inter- 

2 5 rupts to the processor 2 0 when they are activated. 
Operation of the Device in the First Mode 

In the first mode, the device operates as follows. 
Suppose that a user wishes to record text from a sheet 
of paper. In this case, he holds the device against the 

30 sheet in the location where he wishes to start recording 
a character sequence. He presses a button 27 to activate 
the device and passes the latter over the text he wishes 
to record, following the text in the same way as when one 
reads the text. The tracer LED 28 emits a light beam 

35 which facilitates following the lines. When the user 

activates the device, the processor 2 0 commands the LED 6 
to begin generating strobe pulses at a predetermined fre- 
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quency of 2 5 Hz, whereupon the sensor records images with 
partially overlapping contents which are stored in the 
read/write memory 23. However, text which is stored in 
the form of images requires a large amount of memory 
5 space. Consequently, to save memory space in this mode, 
the characters in the images are identified and stored 
with the aid of ASCII code. When the user has passed the 
device over the selected text or has come to the end of a 
line, he lifts the device off the sheet and releases the 

10 activating button, whereupon the processor 20 turns off 
the LED 6. When the selected text has been recorded, the 
user can control the device to show the recorded text 
on the display 25 or to transfer the text to an exter- 
nal computer by the intermediary of the IR transceiver 

15 26. The possibility of showing the scanned information 

directly on the display has proven very important since a 
user often wishes to verify that the correct information 
has been scanned. 

The flowchart in Fig. 3 shows how the device is 

2 0 adapted to work in the first mode. In step 301, images 
with overlapping contents are recorded with the aid of 
the above device and are stored in a current image area 
in the read/write memory 23. The images are stored as 
images, i.e. with the aid of a plurality of pixels, each 

2 5 having a grey scale value in a range from white to black. 

As soon as an image has been stored in the current 
image area, the putting- together of the image with the 
previous image is suitably commenced, step 302 , if such 
an image is available. If there is no previous image, the 

3 0 current image is entered directly into a line image area 

in the read/ write memory. 

In order to determine how the current image is to 
be put together with the previous image so that the best 
match is achieved between the contents of the images, 
3 5 every possible overlap position between the images is 

examined, at the pixel level, and an overlap measurement 
is determined as follows: 
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1) For each overlapping pixel position, the grey 
scale values of the two relevant pixels are added up if 
the latter are not white. Such a pixel position in which 
none of the pixels are white is designated a plus posi- 

5 tion. 

2) The grey scale sums for all the plus positions 
are added up. 

3) The neighbours of each pixel position are exa- 
mined. If an overlapping pixel position is not a neigh- 

10 bour of a plus position and consists of a pixel which is 
white and a pixel position which is not white, the grey 
scale value of the non-white pixel is subtracted, pos- 
sibly multiplied by a constant, from the sum in point 2) . 

4) The overlap position providing the highest over- 
15 lap measurement as stated above is selected. In the 

resulting composite image the mean value of the grey 
scale value of the overlapping pixels is used. In this 
way, noise can be suppressed in the overlap area. The 
putting-together is thus carried out both vertically and 
20 horizontally. If it is detected that, when being put 

together, the images do not end up on a horizontal line, 
the composite image is suitably adjusted so that it 
becomes horizontal, for example by turning the composite 
image . 

25 Our Swedish patent application No. 9704924-1 and 

the corresponding US Application No. 024 641, describe 
an alternative way of matching the images in order to 
find the best overlap position. The content of these 
applications is herewith incorporated by reference. 

3 0 The composite image gradually develops in the line 

image area in the read/ write memory. It is preferred that 
the line image area be big enough to store an A4 line of 
normal typewritten text. 

In step 303, the software of the processor 20 

3 5 divides the composite image in the line image memory 
area into sub- images each containing only one charac- 
ter. The purpose of this is to create input signals to 
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a neural network software which is to interpret the char- 
acters. The division is effected by adding up the grey 
scale values of the pixels for each pixel row and each 
pixel column in the composite image. By studying the 
local intensity minima for the row sums and column sums 
thus obtained, boundaries can be determined for the 
extent of each character in the image. 

Subsequently, in step 3 04, each character in the 
composite image of the character sequence imaged is 
interpreted. The grey scale values of the pixels which 
together constitute a sub- image containing only one 
character are fed as input signals to a neural network. 
Each output from the neural network represents a charac- 
ter which the network can identify. The output from the 
network which has the highest output signal is chosen. and 
the character thus chosen is stored in step 305 using a 
predetermined character code format, for example ASCII 
code, in the read/write memory 23 in a memory area for 
interpreted characters. When the character identification 
and storing in character-coded format is completed, the 
processor activates the indicating device 2 9 to inform 
the user that it is ready to record a new character 
sequence, step 306. Subsequently, it goes back to step 
301 . 

In the first mode, the steps described above are 
thus carried out by the processor 2 0 with the aid of the 
associated units and suitable software. Such software can 
be created by the skilled person with the aid of the 
above instructions. The character recognition is perform- 
ed with the aid of neural network software, which is 
adapted in a suitable manner. Neural network software is 
commercially available from, for example, MATLAB Neural 
Network Toolbox, The MathWorks Inc., 24 Prime Park Way, 
Natick, MA 01760, USA. 

Adjustment of the Device betw een the two Modes 

Suppose that the user now wishes to supplement the 
scanned text with an image of an object. In this case, 
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he indicates this by pressing the button 27. Then lens 
system is then moved to the position B. This movement 
results in the focus of the device being changed so that 
a clear image of an object located at a distance can be 
5 projected onto the light-sensitive surface of the sensor 
8 . 

Operation of the Device in the Second Mode 

In the second mode, the device operates as follows. 
The user directs the window of the device at the object 

10 which is located at a distance and which he wishes to 

image. On the display 25 of the device, the user can see 
the image which will be recorded. When the user is satis- 
fied with the appearance of the image, he presses a but- 
ton 2 7 to activate the device which then records an image 

15 of the object and stores the image in image format in the 
memory 22. When the image of the object has been record- 
ed, the user can control the device to either show the 
recorded image on the display 25 or transfer the image to 
an external computer by the intermediary of the IR trans - 

2 0 ceiver 26. The recorded image can then be supplemented 

with additional text information by the user setting the 
device to the first mode and recording text or other 
characters. Subsequently, the text and image information 
recorded can be shown either on the display 2 5 of the 

2 5 device or on an external computer. 

The flowchart in Fig. 4 shows how the device is 
adapted to operate in the second mode. In step 401, the 
extent of the image is indicated on the display 2 5 of the 
device. When the user is satisfied with the appearance of 
30 the image, he presses the button 27 and, in step 402, the 
image is then recorded. The image is recorded in the cur- 
rent image area of the read/write memory with the aid of 
a plurality of pixels, which can have either grey scale 
values from white to black, or colour values. The user 

3 5 can then choose whether or not he wishes to keep the cur- 

rent image. If the user decides to keep the image, the 
process continues along the solid line to step 403, in 
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which the image is stored in the memory 23 . When the 
image has been stored, the device indicates, in step 404, 
that it is ready to scan a new image. If the user does 
not wish to keep the image, the process continues, from 
5 step 402, along the dashed line back to step 401 in order 
for a new image to be scanned. 
Al t erna t i ve Embodiment s 

The present invention can, of course, be modified 
within the scope of the appended claims. The lens means 

10 may, for example, comprise a lens which is put on like a 
"cap" on the front of the device in one of the modes. 

In the embodiment described above, the device has a 
single light-sensitive sensor which is used in the first 
as well as the second mode. As mentioned above, however, 

15 the device can alternatively have a sensor for each mode. 
Fig. 5 shows schematically how the embodiment in Fig. 1 
could be modified to have two sensors. In the embodiment 
in Fig. 5, the device has a second window 2' in the side 
of the casing, a second light-sensitive sensor 8 1 with a 

20 two-dimensional sensor surface, and a second lens means 
7' which can have a variable focus. The electronic cir- 
cuitry part is the same as in the embodiment in Fig. 1. 
When adjusting from one mode to the other, an adjustment 
takes place of which sensor is activated for imaging and 

25 an adjustment takes place of from which sensor the elec- 
tronic circuitry part collects images. 



